Performance Test Closure Report for <Application Name>
Performance Center of Excellence(PCoE)
TMH
7/15/2014

Contents

1.0	Introduction	3

1.1	Purpose	3

1.2	Distribution List	3

1.3	Document History	3

2.0	Overall Summary	3

2.1	Executive Summary	3

2.2	Performance Test Objectives:	3

2.3	Out Of Scope	3

2.4	Constraints	3

2.5	Assumptions and Dependencies	3

3.0	Test Summary	3

3.1	Performance Test Execution Summary	3

3.2	Performance Test Environment	3

3.3	Performance Test Tools and Monitoring	3

3.4	Performance Issues Identified	3

4.0	Performance Test Results	3

4.1	Load Test:	3

4.1.1	Load Test Results	3

4.1.2	Load Test Observations	3

4.2	Stress Test:	3

4.2.1	Stress Test Results	3

4.2.2	Stress Test Observations	3

4.3	Endurance Test (wherever applicable):	3

4.3.1	Endurance Test Results	3

4.3.2	EnduranceTest Observations	3

5.0	Recommendations and Sign Off	3

5.1	Performance Test Recommendations:	3

5.2	Performance Test Sign Off:	3

6.0	Appendix	3

6.1	Related Documents:	3

6.2	Test execution details and graphs:	3

6.3	Monitoring reports	3

6.4	Sharepoints links	3

7.0	Glossary	3

Introduction

Purpose

This document summarizes the Performance Test Execution details, Results Summary and Closure for the application(s) that is being tested.

Distribution List

The intended audience for this document are:

Solution Architects

Techical Owners

PCoE Team

Document History

Table 1: Document History

Version	Date	Author	Changes
0.1	16-July-2014	Saleem Khan	Initial Draft
0.2	17-July-2014	Saleem Khan	Updated to the Test Results, multiple sections
0.3	18-July-2014	Saleem Khan	Added Executive Summary, accepted the review comments
0.3	31-July-2014	Ashwani Kumar	Validated all the review comments

Table 2: Revision History

Version Reviewed	Date	Reviewer	Reviewer Role
0.2	18-July-2014	Shanafelt, Garrett	Performance Test Manager
0.3	18-July-2014	Shanafelt, Garrett	Performance Test Manager

Table 3: Approval History

Version Approved	Date	Approver	Approver Role
0.3	01-Aug-2014	Shanafelt, Garrett	Performance Test Manager
Kelly, Kevin	QA Head

List of Tables

Table 1: Document History	3

Table 2: Revision History	3

Table 3: Approval History	3

Table 4: NFR traceability	3

Table 5: Defect Details	3

Table 6: Load Test Details	3

Table 7: Stress Test Details	3

Table 8: Endurance Test Details	3

Table 9: Sign Off Summary	3

Overall Summary

Executive Summary

Based on the Performance Test results obtained from the <Application Name> performance testing, it is concluded that the project has met its test objectives which included (refer to below examples):

90 percent of <Application Name> transaction response time were observed to be meeting the SLA

CPU and Memory utilizations were under the acceptable threshold during the peak and the endurance test.

PCoE recommends that the <Application Name> project should proceed with the scheduled next phase

Performance Test Objectives:

The Performance Test Objectives for <Application Name> was to <Performance Goals details>

Performance Test was conducted for the following services <Business Service Names>

Performance test was conducted to identify the performance bottlenecks in the <Interface Details> and tuning of the entire application for achieving the performance goals.

Out Of Scope

List out the items which were out of scope for performance testing stating the reason for exclusion. Refer to the below example:

End to End Testing from ACI user interface was out of scope for Credit Card Enhancement as the ACI QA environment is not a performance environment and not comparable to its production environment. Hence only a nominal load was put via ACI (1/10th of the load volume), the major portion of the load (9/10th of the load volume) was directly injected to the ESB via SOAP requests.

Constraints

Mentions the constraints that were experienced in the performance test. Refer to the below statements as examples:

Resource availability during the execution window was a constraint in issue resolution and test execution information.

FNFG technical knowledge and understanding of the system interactions and their effects on the other systems was an initial constraint. This manifested in access related requests for the various components and looping in technical support for assistance.

Assumptions and Dependencies

List out the assumptions and dependencies for the performance test. Refer to the below examples:

The Peak hour volumes which was used to setup Performance Test and Certification Test target volumes were derived based on the interpretation of the ACI logs from production

The loan accounts used for performance test were Performing Loans (active loans that could accept payments), a subset of the loans provided by the ACBS project team which were not being used for their functional testing.

Test Summary

Performance Test Execution Summary

The Performance test execution for <Application Name> was started on <Date>

The Test execution was brought to closure on <Date>

A Total of <# of Test Cycles> cycles was executed spanning over <Execution Days> days.

Performance Test was conducted during the <Execution Window> on weekdays/weekends subject to environment availability.

Please find below the traceability matrix for Non Functional Requirements tested and the SLA met/not met details

Table 4: NFR traceability

NFR ID	NFR Details	SLA details	Business Service Tested	90% Response Time(sec)	SLA Met(Y/N)
1	The Credit Card Details Should be displayed for retail customers under 4 seconds	< 4 seconds	1.1 Credit Card Details	3.9	Y
1.2 Credt Card Transactions	3.8	Y
3.6	Y
3.8	Y
3.8	Y

During the test execution, there was participation from FNFG technical team and vendor support which are listed below (below list is for reference)

ESB Integration Team

FNFG Infrastructure Architecture Team

Application Development Team

ACI support team

ACBS support team

IBM support team

Miser Support Team

Production Support Team

Please refer to Appendix section<Section Name>for the details of the various execution cycles that were run in the performance test.

Performance Test Environment

The Performance Tests were conducted in the <Environment Name> environment which was configured to point to the following instances (refer to the examples below):

MISER – ACBS MISER

ACI – ACI TEST2 (ACI GUI transactions)

ACBS – FNS

ESB – QA (Classic mode/Active Active mode)

Performance Test Tools and Monitoring

Performance Test was executed using the HP Load Runner <Version Number>.

HP Sitescope <Version Number> was used to monitor the following Application architecture components (refer to the examples below):

ESB Message Queues

Queue Depths and Channel parameters

Cohesion Server

CPU & Memory

Dispatcher Dot Net boxes

CPU & Memory

During initial testing, the MQ Queues were monitored using Queue Depth monitor tool.

NMON was used to collect the server stats for the <Server Name> server

SAR Report was used to collect the server stats for the <Server Name> server

Performance Issues Identified

The Performance test executions cycles were helpful in uncovering many bottlenecks, which were resolved during the course of the testing. Please find below the details:

Table 5: Defect Details

Defect ID	Defect Type	Defect Desciption	Defect Resolution
# from ALM	Critical
High
Medium
Low

Please refer to the ALM/Defect Tracker link for this project to view the detailed defect summary

<ALM or any other Defect Tracker document link>

Performance Test Results

Load Test:

Load tests for <Application Name> were conducted from <Start Date> to <End Date>.

A total of <# of Load Test Cycles> was executed. Please find below the results, graphs and observation details.

Load Test Results

Load tests results

Table 6: Load Test Details

Load Test Details	Resp time SLA met (Y/N)
Test Cycle	Test Iteration	Vusers	Date/time
1	1.1	50	5/15/2014 21:13:36 -5/15/2014 23:13:40	N
1.2	30	N
1.3	50	Y
2	2.1	100	Y
2.2	100	Y
2.3	100	Y

Load Test graphs – Performance Trend graphs.

Load Test Observations

Examples include:

A total of 3953 successful transactions were achieved in the Certification Test execution

90% of the Get Account Information requests for a Miser Checking account had a response time of < 1 seconds

90% of the Get Account Information requests for a Miser Loan account had a response time of about 2 seconds indicating that Miser has no delays in the responses even under load

The response time for Miser related transactions were much better than the ACBS related transactions under the performance load i.e. 60 request threads and approximately 3000 transactions/hour

Stress Test:

Stress tests for <Application Name> were conducted from <Start Date> to <End Date>.

A total of <# of Stress Test Cycles> was executed. Please find below the results, graphs and observation details.

Stress Test Results

Stress test results

Table 7: Stress Test Details

Stress Test Details	Resp time SLA met (Y/N)
Test Cycle	Test Iteration	Vusers	Date/time
1	1.1	500	5/15/2014 21:13:36 -5/15/2014 23:13:40	N
1.2	500	N
1.3	500	Y
2	2.1	500	Y
2.2	500	Y
2.3	500	Y

Stress test Graphs – Performance Trend graphs

Stress Test Observations

Examples include:

A total of 7000 successful transactions were achieved in the Stress Test execution.

There were no spikes in response times ACI UI transaction / XML requests at the ESB.

Endurance Test (wherever applicable):

Endurance tests for <Application Name> were conducted from <Start Date> to <End Date>.Please find below the results, graphs and observation details

Endurance Test Results

Endurance test results

Table 8: Endurance Test Details

EnduranceTest Details	Resp time SLA met (Y/N)
Test Cycle	Test Iteration	Vusers	Date/time
1	1.1	100	5/15/2014 18:13:36 -5/15/2014 23:13:40	N
1.2	100	N
1.3	100	Y
2	2.1	100	Y
2.2	100	Y
2.3	100	Y

Endurance test Graphs – Performance Trend graphs

EnduranceTest Observations

Examples include:

The Endurance Test was run for 5 hours with the 100 Vuser load.

A total of 2200 transactions were executed in the Test Duration.

No memory leaks were observed in the test. The Average response time was consistent throughout the test after the initial ramp up.

Recommendations and Sign Off

Performance Test Recommendations:

List out the recommendations based on the overall performance test executions results.

Highlight the potential problems and challenges that could be encountered in production.

Performance Test Sign Off:

The <Application name> Performance testing activities have concluded with the successful completion of all the planned performance test cycles.

The Performance test results are acceptable to the business and application team. The results have been reviewed and approved.

Table 9: Sign Off Summary

Authority	Name	Title/Team	Sign Off Email
Sign Off required	Solution Architect	<Attach the sign off email>
Sign Off required	Project Manager
Sign Off required	Application Owner
Review/Approve	Performance Manager

Appendix

Related Documents:

Test execution details and graphs:

Monitoring reports

Sharepoints links

Glossary

PCoE	Performance Center of Excellence	PCoE is an enterprise framework to establish and enable strong quality processes for Performance testing. It comprises of strategic assets (people, process and technology) tied together to find the stability and reliability of an application. The PCoE framework leverages tools, techniques, and methodologies to deliver technology better, faster and cheaper.
NMON	Nigel’s Monitor	NMON is used for linux system monitoring.
PerfMon	Performance Monitor	PerfMon is used to monitor windows based applications
ALM	Application Lifecycle Management	HP tool used for test management in FNFG.
SLA	Service Level Agreement	A Service Level Agreement (SLA) is a part of a service contract where a service is formally defined.
SAR	System Activity Report	System Activity Report is a Solaris-derived system monitor command used to report on various system loads, including CPU activity, memory/paging, device load, and network. Linux distributions provide SAR through the system statistics package.